fix(googlechat): supported keyless ADC and reliable send-once - #1513
fix(googlechat): supported keyless ADC and reliable send-once#1513chaodu-obk[bot] wants to merge 13 commits into
Conversation
Keyless ADC (MetadataTokenSource): mint a chat.bot-scoped token from the workload's own GCP identity — GCE metadata (SA email + base token) -> IAM Credentials generateAccessToken (self-impersonation). No SA key file. Config [googlechat].use_adc / GOOGLE_CHAT_USE_ADC; auth precedence SA key > ADC > static token; cache under the IAM-granted expireTime (fallback 3600s). Send-once for Google Chat: its write rate limit is 1/sec/space (create+patch+delete combined) so per-token streaming edits 429, and the unified adapter returns a synthetic message id that patch can't target (404). googlechat added to NON_STREAMING_PLATFORMS (renamed from NON_EDITABLE_PLATFORMS); resolve_streaming forces send-once on both the embedded dispatch (stream_prompt_blocks) and WebSocket gateway paths. Also: Dockerfile.claude OPENAB_BUILD_FEATURES arg, Helm googleChat.useAdc value, docs + config-first conformance entry + googlechat.toml schema record. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- F15: bound every token-mint request (SA-key exchange, metadata, IAM Credentials) with a 10s TOKEN_REQUEST_TIMEOUT so a hung connection cannot stall senders behind the cache write lock or defeat the ADC -> static token degradation path - F16: reject empty/whitespace minted tokens at all three extraction sites (SA-key exchange, metadata base token, generateAccessToken) so a malformed response follows the degradation path instead of being cached as valid - F17: correct the shorthand precedence wording in config.toml.example, config.rs, config-reference.md, google-chat.md env table, and values.yaml to name the configured-but-unloadable-key -> ADC fallback - F19: refuse edit_message for non-resource-name (synthetic unified_) ids locally instead of sending a doomed patch (400 INVALID_ARGUMENT) - F20: cross-reference the two sibling streaming gates (resolve_streaming / platform_supports_streaming) in both docs - F21: document get_token precedence and its asymmetric failure behavior at the function - F22: replace from_parts' five positional args with a named GoogleChatParts struct; all call sites and tests name their fields - F23: install metadata_source only when no SA key loaded, so the code encodes the precedence it documents - F24: drop private review-numbering labels (F1/F2/F4/F5) from source comments and test comments - F25: fix the self-contradictory 'immutable after creation' GCE scope wording in docs/google-chat.md Option C - F26: identify the orphaned Secret (agentFullname convention + discovery commands) in the key-to-ADC migration note - F27: log the resolved service-account identity on successful mint - F28: classify generateAccessToken failures (insufficient_scope / missing_role / api_not_enabled) in the error string New regression tests: loaded-key-suppresses-ADC-source, blank-minted- token rejection (wiremock), synthetic-id edit_message no-op (wiremock, expect(0)), and error-classification table.
Route delete_message with the other unsupported Google Chat commands so it returns before token resolution, logging, or network work instead of falling through to the empty-send response path. Add a regression test that distinguishes the old fallthrough behavior and update the platform schema feature/quirk notes to document the explicit no-op.
Share one non-whitespace token validator across the SA-key, metadata, and
IAM response paths and cover empty/whitespace/valid values in a table test.
Require exact spaces/{space}/messages/{message} edit targets, cover malformed
resource shapes, and assert the valid edit path issues exactly one PATCH.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Important
CHANGES REQUESTED
Consolidated review: #1513 (comment)
GitHub event: COMMENT - self-review delivery only; this is not an approval.
- require a distinct adc_target_service_account for keyless ADC; reject runtime/target equality before requesting the metadata base token because Google prohibits access-token self-impersonation - plumb GOOGLE_CHAT_ADC_TARGET_SERVICE_ACCOUNT through config, env, Helm, docs, schema, and config-first conformance; log both runtime and target SAs - classify documented FAILED_PRECONDITION self-impersonation errors - decouple normal-reply acknowledgements from cosmetic streaming so Google Chat remains send-once but carries/awaits request_id; promised ack failures, channel closure, and timeout now fail closed instead of reporting gw_sent - remove unverifiable ADC/static-token identity-equivalence claims and log static fallback as a possible identity switch - add 30s failed-refresh cooldown so queued senders reuse a still-valid token instead of serially repeating metadata/IAM timeouts - add regression tests for distinct-target enforcement, ack error propagation, and refresh retry suppression
This comment has been minimized.
This comment has been minimized.
The MetadataTokenSource client carries the plaintext GCE metadata bearer. Two hardening fixes on its construction: - Add .no_proxy(): reqwest honors HTTP(S)_PROXY by default, which could route the metadata access-token response through an operator/attacker proxy hop. Disable proxies so the bearer never leaves the metadata path. - Replace .build().unwrap_or_default() with .expect(): on a builder error unwrap_or_default() yields a DEFAULT client that follows redirects and honors proxies, silently defeating the no-redirect/no-proxy guarantee this source exists to uphold. Fail loud at construction instead. Addresses supplementary review findings on #1513 (F3/F4).
Brings in the Hermes removal (#1523) so this branch no longer builds or CI-tests the Hermes image — the flaky raw.githubusercontent.com 429 smoke failures were unrelated to this PR. No conflicts (ADC changes and the Hermes deletions touch disjoint regions of values.yaml / config-reference.md).
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Close the remaining review gaps in the corrected keyless Google Chat flow: - share a direct Result-returning delivery path between standalone WebSocket handling and the unified in-process adapter, so unified auth/API failures reach core instead of becoming synthetic success; preserve real message IDs - bound the complete delivery to 30s (inside core's 35s ack window), add 10s per-mutation timeouts, and stop on the first failed chunk with explicit partial-delivery context - validate/normalize user-managed target SA emails before IAM URL construction, rejecting numeric unique IDs and path/query/trailing-dot aliases; keep the MetadataTokenSource constructor private behind that invariant - attach an existing ServiceAccount to gateway pods with precedence gateway > per-agent > global, document GKE Workload Identity setup, and add Helm tests (including numeric-looking KSA names rendered as strings) - classify the proxy-routing test as ignored integration, accept common GOOGLE_CHAT_USE_ADC case/whitespace forms, and avoid false failures when an accepted 2xx response omits a message resource name Validated on macmini: workspace all-feature check, gateway+unified clippy with -D warnings, gateway tests (332 pass/1 ignored), ignored proxy test explicitly, unified/core regressions, feature-off core check, and Helm unittest 46/46.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Important
CHANGES REQUESTED
Consolidated review: #1513 (comment)
GitHub event: COMMENT - self-review delivery only; this is not an approval.
Close the final review gaps in standalone Google Chat delivery: - spawn Google Chat handling immediately from the WebSocket receive loop so replies no longer wait behind an inline send while core's ack clock runs - serialize actual sends with an adapter-owned delivery mutex, starting the absolute 30s deadline before lock acquisition so queue + token + chunks are all inside the 35s core acknowledgement window - replace the context-free outer timeout with phase/chunk-aware deadline waits; queue, token, and partial-chunk timeout errors now retain delivery context - add queue-wait and delayed partial-timeout regressions - classify every PR-added loopback-network/filesystem test as ignored integration and run all ten explicitly in validation Validated on macmini: workspace all-feature check, gateway/unified clippy with -D warnings, 324 normal gateway tests + 10 ignored integrations, and unified failure propagation regression all pass.
This comment has been minimized.
This comment has been minimized.
|
Correcting the record on the self-impersonation question, since I argued the other side on #1512 and that argument was wrong. I was wrong, and specifically about the reasoning — not only the conclusion. My earlier position was that the flow was acceptable because the runtime SA holds
The criterion is the caller's credential type plus caller-identity == target-identity. Scope is not part of it. My argument reconciled "the docs say no" with "it works in practice" by inventing a mechanism, and I stated that invention as fact. That was the error. On the empirical observation. It remains true that the flow minted What we did about it. We adopted this PR's design in production rather than continuing to argue for ours:
A design point that settles this independently of the contract. Self-impersonation cannot express a cross-project target at all: the adapter can only mint for the identity the metadata server returns. Any deployment whose Chat app SA lives in a different project requires this PR's distinct-target model. We hit exactly that constraint, which is what pushed us onto this design before I had re-read the documentation. I'm not commenting on the currently open blockers — those are the maintainers' call. Happy to supply further detail from the production deployment if it is useful for verification. |
|
Note LGTM ✅ - All frozen Review Contract criteria are resolved at exact head What This PR DoesThis PR adds supported keyless Google Chat authentication through a runtime service account impersonating a distinct Chat-app service account. It keeps Google Chat send-once while preserving delivery failure reporting across standalone and unified paths, and hardens credential boundaries, timeouts, Helm configuration, diagnostics, documentation, and regression coverage. How It WorksThe gateway validates a distinct target service-account email, reads the attached runtime identity and metadata credential, and calls IAM Credentials Findings
Resolution DetailsF1: Construction-time PEM validation
F2: Helm upgrade identity compatibilityThe gateway template now reads only the explicit gateway ServiceAccount value. Helm tests prove that chart-global and per-agent values do not affect the gateway, while explicit gateway configuration, Workload Identity pairing, and numeric-looking names still render correctly. README, values, config reference, and Google Chat docs match the behavior. F3: Missing-adapter failure acknowledgementThe production WebSocket dispatch seam now routes missing-adapter failures through the same structured response helper used by configured delivery. The response preserves Addressing External Reviewer Feedback@sebastian-hsu
Accepted and corroborated: this evidence supports the selected two-identity architecture. The current exact head retains the distinct-target guard before base-token retrieval. The reported production success for cross-project impersonation and configured delivery acknowledgement is useful operational evidence and introduces no new blocker. All six existing inline review threads are now resolved. Accepted residual risks and Follow-ups remain explicitly non-blocking under the frozen PR contract. Baseline Check
Validation
Independent Review Coverage
What's Good (🟢)
5. Three Reasons We Might Not Need This PR
These trade-offs are accepted and do not violate the frozen Review Contract. |
Summary
This PR carries #1512 plus focused maintainer fixes for Google Chat authentication and delivery reliability.
It adds supported keyless authentication with two distinct service accounts: the attached runtime service account obtains a metadata credential and impersonates a dedicated Google Chat service account for the
chat.botscope. Runtime/target equality is rejected before the base token is requested because Google prohibits access-token self-impersonation.Google Chat remains send-once, but delivery acknowledgement is independent from cosmetic streaming. Configured standalone and unified paths report API, authentication, channel, and timeout failures to core instead of returning synthetic success.
Architecture
Keyless ADC
generateAccessTokenfor the distinct target withchat.bot.The runtime identity needs
roles/iam.serviceAccountTokenCreatoron the target service account. Existing service-account-key and static-token paths remain available. Auth precedence is: a successfully loaded service-account key, then ADC, then an explicitly configured static token.Send-once delivery acknowledgement
request_idfor Google Chat normal replies even though cosmetic streaming is disabled.Resultdirectly; the standalone path receives a correlatedGatewayResponse.Review Contract
This section is the exclusive merge-review contract for this PR. Review findings must map to one of the blocker rules below. A useful observation that does not map to a blocker rule is a non-blocking follow-up and must not extend the finish line.
Goal
Deliver a supported two-identity keyless Google Chat auth path and reliable send-once failure reporting without removing existing auth methods or changing unrelated platforms.
In scope
Non-goals
GoogleChatTokenCacheandMetadataTokenSource; tracked by Consolidate googlechat token-source cache machinery (review F29 follow-up) #1514.Exclusive blocker rules
A finding blocks this PR only when at least one rule is satisfied:
Every blocking finding must cite its rule, affected supported path, concrete evidence, and a testable requested change. Architecture preference alone is not a blocker.
Non-blocking follow-ups
The following do not block when the implementation is correct under the criteria above:
Non-blocking observations should be recorded once, deduplicated, and moved to a follow-up rather than causing another review round.
Accepted Residual Risks
The owner accepts these for this PR; they are follow-ups unless evidence shows a blocker-rule violation:
Retry-Afterand per-space pacing are follow-ups.unclassifiedwith a truncated raw body.Acceptance Criteria
Completed:
0d142de33204d98169d7171926a35b9b3d68c38dpasses all required checks: 39 success and one intentional skip.Resolved frozen blockers:
request_idreceives an immediate structured failure when the Google Chat adapter is unavailable.Follow-ups
These are explicitly non-blocking for this PR:
Scope freeze and review closure
3771053f;0d142de3fixes the only CI regression from that delta.f59cbf9c.Current review state
The prior broad review reported six important items. Under this bounded contract:
3771053f, with the Clippy regression fixed in0d142de3.The final exact-head focused review is ready at
0d142de33204d98169d7171926a35b9b3d68c38d.Commits
bd62ee49- resolves timeout, token-boundary, precedence, documentation, readability, and operability findings.904f626c- makes delete an explicit no-op.fa9d58a9- locks token and edit-resource boundaries.3a1ce860- implements distinct-target ADC, delivery acknowledgement, fallback diagnostics, and ADC refresh cooldown.3b08312- hardens metadata transport with no-proxy and fail-loud construction.61153772- separates metadata/IAM proxy policy, validates static-token boundaries, and improves ADC diagnostics.31cea79d- propagates unified delivery outcomes, bounds delivery, validates target emails, and wires gateway ServiceAccounts.b298e0a8- includes queue wait in the deadline and preserves partial-timeout context.f59cbf9c- resolves the owner-approved review NIT batch.3771053f- resolves the three frozen contract blockers with focused tests and docs.0d142de3- explicitly detaches the Google Chat delivery task to satisfy Clippy.Validation
At exact head
0d142de33204d98169d7171926a35b9b3d68c38d:git diff --check; OpenSSL validated the 2048-bit RSA fixture; focused correctness/test audits returned LGTM; exact-head CI compiled and exercised the Rust and Helm changes.This validation records the current head only. Any subsequent code change creates a new exact head and must rerun the required checks.